[WIP] Ship BR schema as standalone llmdbenchmark-schema package#1273
Draft
Bslabe123 wants to merge 1 commit into
Draft
[WIP] Ship BR schema as standalone llmdbenchmark-schema package#1273Bslabe123 wants to merge 1 commit into
Bslabe123 wants to merge 1 commit into
Conversation
The Benchmark Report (BR) schema types are useful to external workload
generators that produce BR reports (e.g. inference-perf) and to external
analysis pipelines that consume them. Today, taking a dependency on these
types means depending on the full llmdbenchmark distribution, which pulls
in kubernetes/transformers/huggingface_hub/etc. External tools typically
have none of these in their dependency closure, so they either vendor the
schema files or skip the typed API entirely.
This change splits the schema into its own pip-installable distribution
(llmdbenchmark-schema) sitting in the same repo, with the same maintainers,
published from the same CI workflow. The full llmdbenchmark package depends
on it and re-exports the types under llmdbenchmark.analysis.benchmark_report
so existing callers do not need to change.
Layout:
llmdbenchmark-schema/
pyproject.toml (name = "llmdbenchmark-schema")
llmdbenchmark_schema/
__init__.py
base.py (moved from analysis/benchmark_report/)
schema_v0_1.py (moved)
schema_v0_2.py (moved)
schema_v0_2_components.py (moved)
br_v0_*_example.yaml (moved)
br_v0_*_json_schema.json (moved)
py.typed
README.md
llmdbenchmark/
analysis/
benchmark_report/
__init__.py (re-exports from llmdbenchmark_schema)
cli.py, core.py, native_to_br0_*.py (import from llmdbenchmark_schema)
pyproject.toml (adds llmdbenchmark-schema as a dep)
Runtime deps for llmdbenchmark-schema: pydantic>=2.0, PyYAML.
Verified locally:
- pip install -e llmdbenchmark-schema/ succeeds
- from llmdbenchmark_schema import BenchmarkReportV02 works
- br_v0_2_example.yaml round-trips against the schema
- All updated files under llmdbenchmark/analysis/benchmark_report/ compile
Open questions / WIP items:
- Wire the publish workflow to ship both distributions from the same CI run.
- Version policy: llmdbenchmark-schema is set to 0.2.0 (tracks the latest
BR schema version it ships). Should it follow llmdbenchmark's versioning
instead?
- Sweep for any other places in the repo still doing relative imports of
the moved files.
Motivation for inference-perf:
kubernetes-sigs/inference-perf#461
inference-perf currently vendors ~1300 lines of the BR0.2 schema; this
change would let it depend on llmdbenchmark-schema instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Splits the Benchmark Report (BR) schema types out of the main
llmdbenchmarkdistribution and into a separate pip-installable package,llmdbenchmark-schema, living in the same repo underllmdbenchmark-schema/.External tools that produce or consume BR reports (workload generators like inference-perf, dashboards, analysis pipelines) currently have to either vendor the schema files or pull in the full
llmdbenchmarkpackage, which carrieskubernetes,pykube-ng,kubernetes-asyncio,transformers,huggingface_hub, etc. Those deps are unrelated to the schema and don't belong in the dependency closure of a typed schema consumer.After this change:
pip install llmdbenchmark-schemapulls in onlypydantic>=2.0andPyYAML.pip install llmdbenchmarkcontinues to work and re-exports the schema types underllmdbenchmark.analysis.benchmark_report, existing callers do not need to change.Marked WIP because the publish workflow still needs to be updated to push both distributions from the same CI run (see open questions below). The code split itself is complete and locally verified.
Layout
History is preserved via
git mvso blame and per-file history follow the files to their new location.Verified locally
pip install -e llmdbenchmark-schema/succeeds.from llmdbenchmark_schema import BenchmarkReportV02works.br_v0_2_example.yamlround-trips againstBenchmarkReportV02.llmdbenchmark/analysis/benchmark_report/compile cleanly with the new import paths.Open questions for reviewers
llmdbenchmarkandllmdbenchmark-schemato PyPI from the same CI run.llmdbenchmark/analysis/benchmark_report/; happy to look at more if reviewers point them out.llmdbenchmark-schema(PyPI name) andllmdbenchmark_schema(Python package). Open to bikeshed.Motivation
kubernetes-sigs/inference-perf#461 currently vendors ~1,300 lines of the BR0.2 schema from this repo because there's no other way to consume the types. With this change, that PR can replace the vendored files with
llmdbenchmark-schemaas apipdependency.Filing this WIP as a starting point for discussion; happy to iterate on the structure, naming, or split boundary.